Human Activity Recognition for Synthesis

Given a monocular video input which contains a moving human being, our goal is to automatically recognize the action and then track the articulated pose. Distinct from other example-based approaches such as estimating poses from still images [7] and using multiple cameras [6], our system uses a single camera to reconstruct human motions, combining knowledge from images and motion capture data together. Other model-based motion reconstruction systems are presented in [3, 4, 8]. Figure 1 illustrates the structure of our activity recognition and reconstruction system. We first create a motion capture database containing multiple subjects and action types, and then render the silhouettes from various view points (step A). Based on the training images, we build a hierarchical reliable-inference (RI) classifier [2] (step B). Once the preprocessing is done, we use the RI framework to classify observed motion into known actions by sequentially evaluating posterior class ratios (step C). Next several matches are picked from the database for each test image, based on image features, estimated pose configuration and camera angle transitions (step D). A dynamic programming technique is used to generate the sequences of natural and complex human motion from the database (step E). The pose parameters are further refined [1] to recover global translation and body orientation and improve the match with the original sequence (step F).